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GAO BRIEFING TO THE TASK FORCE REPORT 
ON FINDINGS IN THE INVESTIGATION INTO 
THE FLORIDA-13 CONGRESSIONAL DISTRICT 
CONTESTED ELECTION 


FRIDAY, FEBRUARY 8, 2008 

House of Representatives, 

Task Force on Florida-13, 
Committee on House Administration, 

Washington, DC. 

The task force met, pursuant to call, at 10:09 a.m., in Room 
1310, Longworth House Office Building, Hon. Charles A. Gonzalez 
(chairman of the task force) presiding. 

Present: Representatives Gonzalez, Lofgren, McCarthy, Ehlers 
and Lungren. 

Staff Present: Liz Birnbaum, Staff Director; Thomas Hicks, Sen- 
ior Election Counsel; Janelle Hu, Election Counsel; Jennifer Daehn, 
Election Counsel; Matt Pinkus, Professional Staff Member/Parlia- 
mentarian; Kyle Anderson, Press Director; Kristin McCowan, Chief 
Legislative Clerk; Daniel Favarulo, Legislative Assistant, Elections; 
Gineen Beach, Minority Election Counsel; and Bryan T. Dorsey, 
Minority Professional Staff Member. 

The Chairman. I will call to order at this time the Committee 
on House Administration’s Florida-13 Task Force, and I apologize 
for the slight delay. But we were having problems getting over 
here, obviously, with traffic and such. So my apologies. 

We are going to attempt to be brief today and hear the final re- 
port by GAO. But first I think that it would be appropriate for me 
to extend my thanks, of course, to my fellow members of the task 
force; and that would be Congresswoman Zoe Lofgren and Kevin 
McCarthy from California. I also want to extend my thanks to Con- 
gressman Dan Lungren, who was not an official member of the 
task force but, nevertheless, attended some of the meetings and of- 
fered some very constructive advice which actually was followed. 
And so, Dan, I want to thank you. 

To the majority and minority staff, again thank you. Because, at 
the end of the day, of course, all of the work is truly predicated on 
the product that they produce for us as we go through the machi- 
nations of the task force. 

And definitely to the late Congresswoman Juanita Millender- 
McDonald — and you see her portrait in the back of the room — for 
her fine service and the individual who actually created this task 
force with the simple charge of: Get it right and get to the truth, 
which hopefully that is what we have done. 

( 1 ) 
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I also want to extend the task force’s and the committee’s sincere 
thanks to the Government Accountability Office for their fine work. 
I think when we first selected them, they entered it with some res- 
ervation. 

I think the concern of any agency or department many times is 
that Congress or Members of Congress will drag them into a polit- 
ical debate, which would not serve, of course, their mission nor 
their purpose and would jeopardize their objectivity. That did not 
occur, and we appreciate that they were able to assist us to the de- 
gree and the fine manner in which they did. 

I will be making an opening statement. Then I will be recog- 
nizing the members of the task force for opening statements. 

Clive Thompson, writing in the January 6, 2008, issue of the 
New York Times magazine in an article entitled “Can You Count 
on These Machines?” makes the following observation: “The mis- 
trust of touch-screen machines is thus equal parts technological 
and ideological.” Technology is subject to empirical analysis. Ide- 
ology is not. So today we address that which we are capable of re- 
solving through the scientific method, that is, the technological 
mistrust surrounding the November, 2006, election for the office of 
the United States Representative from Florida’s 13th Congressional 
District. 

The challenge presented before the task force was to determine 
the merit and validity of the central allegation contained in con- 
testant’s notice of contest that the electronic voting machines in 
Sarasota County malfunctioned, resulting in 18,000 undervotes, 
thus bringing into question the reliability of the vote totals deter- 
mining the winner of the election as reported by the Florida State 
officials to the United States House of Representatives. 

The task force at the outset decided that it would not entertain 
testing and findings by opposing experts designated by the contest- 
ant and contestee, thus avoiding a “dueling experts” dilemma. 
Rather, the task force unanimously decided that the United States 
Government Accountability Office, with its credentialed resources, 
would serve as the impartial and independent expert. 

The task force further agreed that it would abide by the GAO’s 
findings barring some substantial basis to question those findings. 

Today, the Government Accountability Office will formally 
present its final report. 

Following established task force procedure, GAO did make a pre- 
liminary briefing before the task force on February 6, 2008, of its 
draft report. 

I want to make an important point. The task force has respected 
GAO’s established procedures and protocols. Until a final draft is 
prepared and presented, GAO’s draft report remains a work in 
progress. On February 6, at the preliminary briefing, task force 
members determined that the contestant and the contestee or their 
representatives should have a copy of the draft report. This was to 
allow them to review the materials and convey any questions or 
concerns to the task force in preparation for today’s meeting. 

I am aware that the draft report was made publicly available, 
contrary to the spirit of cooperation and respect for GAO’s proce- 
dures and protocol. I apologize to the United States Government 
Accountability Office for this was not what I preferred and was 
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contrary to what I stated and requested at the February 6th brief- 
ing. This development is not conducive to creating and maintaining 
a healthy working relationship between Congress and GAO based 
on trust and mutual respect. 

With that, I will recognize the minority member of the task force, 
Congressman McCarthy, for an opening statement. 

Mr. McCarthy. Well, thank you, Chairman Gonzalez. 

Before we begin, I do want to thank the GAO for the work they 
have done. 

I also notice Kurt Browning, Secretary of State of Florida is here 
and thank you for the work you have done. You did the study be- 
fore; and, as GAO went down, they were very complimentary of 
how you opened it up and made sure as we went all the way 
through. 

And I want to thank this committee and especially Chairman 
Gonzalez, because I think you have laid the groundwork of how 
contested races in the future should be handed out. I thought every 
decision or every vote we came to a conclusion was in a bipartisan 
manner. 

And going to the GAO was something new. And with my own ex- 
perience, actually having been a staffer and worked on contested 
races before, I think this is one that has been more thoroughly ana- 
lyzed than any one before it. So I do appreciate the work you have 
done and all the committee, as well as Zoe Lofgren; and I thought 
everybody worked in a bipartisan manner. So thank you for that. 

Today we will hear the report of the GAO’s investigation on this 
most studied election. The results are as clear as we can objectively 
expect. After thoroughly combing and analyzing past studies and 
conducting its own studies to confirm past results, there is no evi- 
dence suggesting any malfunction of the processing and counting of 
ballots on the DREs. 

We will hear from the GAO today and ask about its report that 
concludes that the machines counted votes accurately. With this 
final objective confirmation, I hope that we can finally put to rest 
for the people of the 13th District of Florida the notice of contest 
that the challenger, Christine Jennings, filed asking the House to 
overturn the election results against Congressman Vern Buchanan 
and send a strong message to the American people that results of 
the 2006 election and this particular election have been and were 
always correct. And I look forward to hearing the rest of the report. 

The Chairman. Thank you very much. 

And the Chair will recognize Ms. Lofgren. 

Ms. Lofgren. I would just add that I think I agree with the 
other members of the task force. This has been an endeavor that 
has been bipartisan. I think all of our votes have been unanimous. 
We entrusted the analysis to experts at the GAO, and we didn’t 
interfere with them in any way. We let them go through their 
study, and now they have reached a conclusion. 

And I would only note that if we had had, as the GAO has noted, 
if we had had a paper trail, we would have certainly not had to 
go through this exercise. So this will never happen again in Flor- 
ida, because Florida, pursuant to the Governor’s direction I think 
in the legislature, no longer uses these machines and there will al- 
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ways be a paper trail. So this will be the last such contest we will 
see from the State of Florida. 

And, hopefully, as States change their election systems, we will 
never have this type of situation again. 

I would like to thank again the staff, members of the committee 
and especially you, Mr. Chairman, for your terrific leadership in 
this not-an-easy endeavor. But I think we have done our duty to 
the institution fairly and under the Constitution. 

Thank you. 

The Chairman. Thank you very much. 

And, even though not a formal member of the task force, but 
members, of course, of the full committee to which we will be mak- 
ing a recommendation, hopefully, at the end of this meeting today, 
I do want to recognize for brief opening statements, and that would 
be the ranking member of the full committee, Mr. Ehlers. 

Mr. Ehlers. Thank you, Mr. Chairman; and the majority of the 
statement that I want to say is to commend you and the other 
members of the task force. 

It is obvious to me that your judicial experience has been invalu- 
able to you and that your judicial demeanor in handling this has 
been exemplary, and I deeply appreciate it. I have served on sev- 
eral task forces and chaired one. I know how difficult the task can 
be. And you have been very fair, thorough and evenhanded in the 
handling of this. You have set an example for all the task forces 
of the future, and I really commend you for that. 

The second comment I would like to make is a very important 
side benefit of what we have gone through with this. The use of 
the GAO and their work done on this, I think has set a good exam- 
ple not just for us but for the Nation. It will restore the confidence 
of the public in the process. 

It was not a coincidence that four of the contests were filed from 
Florida. There was only one contest that was filed outside of Flor- 
ida, and that had to do with residency, not machines. But four, the 
other four were filed from Florida, and they questioned the ma- 
chines. Obviously, because of the experience Florida has had over 
the past decade, there is a spillover here. The public has lost con- 
fidence in the results. And I think the result of this is going to 
demonstrate to the public that the voting process, by and large, is 
proper, correct and believable and that the public should not ques- 
tion the accuracy of election results as they have done for the past 
8 to 12 years. 

So you have set an example and a standard that is very helpful 
to the Nation as well as in resolving this case, and I thank you for 
it. 

The Chairman. Thank you very much for your kind words. 

And the Chair will recognize Mr. Lungren for some brief opening 
remarks. 

Mr. Lungren. Thank you very much, Mr. Chairman. 

First of all, thank you for your indulgence in allowing me to par- 
ticipate in the hearings of this task force although I was not a 
member. 

Some more than 20 years ago I was a member of the full body 
when there was a contested election, the results of which caused 
bitterness in the House; and partisanship was exacerbated. I genu- 
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inely did not want us to return to those days and was most inter- 
ested in how this contest would be handled. 

And while I did not know the chairman well before the hearings, 
I did serve with his father and enjoyed my relationship with his 
father and only knew the chairman by reputation. But I want to 
thank you not only on my behalf but on behalf of all Members for 
the way that you conducted this task force. This was somewhat of 
a test of our House as to how we would handle this, not only be- 
cause there were questions raised about the election results but the 
nature of the contest. And I think your judgment in having GAO 
get involved, which was adopted unanimously by the other mem- 
bers of the task force, was very, very essential in the manner in 
which this was handled. 

And while I know we are still awaiting the report, I have looked 
at the draft report; and I think I know what they are going to tell 
us. And I think this does help restore confidence in the ability of 
this House to be able to handle touchy matters like this, but, also, 
it is an example to the American people of where we can work in 
a bipartisan basis, where it is easy to be partisan otherwise. 

And I thank the chairman and the other members of this task 
force. 

The Chairman. Thank you very much, Mr. Lungren. 

The Chairman. At this time, we’ll proceed with the final report 
by the representative from the United States Government Account- 
ability Office. 

STATEMENT OF NABAJYOTI BARKAKATI, PH.D., ACTING CHIEF 

TECHNOLOGIST, APPLIED RESEARCH AND METHODS, GOV- 
ERNMENT ACCOUNTABILITY OFFICE 

Mr. Barkakati. Thank you, Mr. Chairman. 

Chairman Gonzalez, members of the task force, we have sub- 
mitted the formal statement for the record, and I am going to sum- 
marize briefly. 

As our statement says I am here today to present the findings 
on our Florida-13 review, based on the testing we conducted on 
iVotronic voting machines used in the 2006 general election in 
Sarasota County, Florida. 

I would like to begin by thanking the task force for its overall 
support of our efforts and specifically for the assistance provided in 
obtaining the resources of the House recording studio, which were 
critical in successfully completing our tests. 

At the October 2, 2007, meeting of the task force, we proposed 
and you asked us to proceed with three tests — firmware verification 
test, ballot test and calibration test — in order to obtain increased 
assurance that the iVotronic voting machines did not contribute to 
the large undervote observed in the 2006 elections in Sarasota 
County. 

To conduct the three tests, we developed test protocols and de- 
tailed test procedures. We met with officials from the Sarasota 
County Supervisor of Elections and the Florida Department of 
State and Florida Division of Elections in order to make necessary 
arrangements to obtain access to the voting machines and to sched- 
ule and conduct the tests. 
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Thanks to everyone’s cooperation and help, we have been able to 
successfully complete our testing during November 26th through 
December 4, 2007, in Sarasota County’s voting equipment facility 
in Sarasota, Florida. 

Our tests on the randomly selected iVotronic machines in Sara- 
sota County did not identify any problems that would indicate that 
the machines were responsible for the undervote in the Florida-13 
race in the 2006 general election. 

In our firmware verification test, we extracted the firmware from 
a random sample of 115 iVotronics and found that in each case the 
firmware extracted from the machines matched the firmware that 
was escrowed and certified by the Florida Division of Elections. 

The statistical approach that we used in selecting these ma- 
chines enables us to say with a 99 percent confidence level that no 
more than 60 of the 1,499 iVotronics that were used in the 2006 
election could have been running a different version of software. 
Consequently, we were able to place more confidence in the tests 
that we conducted on a smaller number of machines, both we con- 
ducted as well as the ones that were conducted by others in the 
past, the results of which have indicated that the iVotronics did not 
cause the undervote. 

Prior to the Sarasota testing, on November 19, 2007, we had vis- 
ited the manufacturer of the voting machine, ES&S, in their Rock- 
ford, Illinois, facility and observed as they rebuilt the firmware 
from the source code that was previously held in escrow by Florida 
Division of Elections. The software that was rebuilt from that 
source code, we observed that that software matches the firmware 
that was held in escrow and that was certified by the Florida Divi- 
sion of Elections. This provides further confidence in the prior 
source code reviews that were conducted by a team from Florida 
State University and by us. 

For the ballot testing, we cast predefined test ballots on 10 
iVotronic machines and confirmed that each ballot was displayed 
and recorded accurately. The test ballots represented 112 ways a 
voter may have interacted with the iVotronic to cast a ballot in the 
Florida-13 race. These test ballots were cast on nine machines that 
were configured as election day machines and repeated on one ma- 
chine configured as an early voting machine. 

Finally, we conducted the calibration testing by miscalibrating 
two of the iVotronics and casting some of the test ballots on them. 
Our tests, involving 10 different miscalibration patterns and cap- 
turing a total of 39 votes on the two machines, found that, al- 
though the machines became more difficult to use with 
miscalibration, the selections that were displayed on the screen for 
the Florida-13 race, were the same ones that appeared on the re- 
view screen and then recorded when the ballots were cast. 

Based on the testings that we have conducted, we have obtained 
increased assurances that the iVotronic voting machines used in 
Sarasota County’s 2006 general election did not contribute to the 
large undervote in the Florida-13 contest. Although the test results 
cannot be used to provide absolute assurance, we believe that these 
test results, combined with the other reviews that had been con- 
ducted by the State of Florida, us and others, have significantly re- 
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duced the possibility that the iVotronic machines were the cause of 
the undervote. 

At this point, we believe that adequate testing has been per- 
formed on the voting machine software to reach this conclusion, 
and we do not recommend further testing in this area. 

Given the complex nature — complex interaction of people, process 
and technology that must effectively work together to achieve a 
successful election, we acknowledge the possibility that the large 
undervote in the Florida-13 race could have been caused by factors 
such as voters who intentionally undervoted or voters who may not 
have properly cast their ballots on the iVotronic machine poten- 
tially because of issues relating to interaction between the voters 
and the ballot. 

Mr. Chairman, this concludes a summary of my written state- 
ment. I would be happy to answer any questions that you or other 
members of the task force may have at this time. 

Thank you. 

[The statement of Mr. Barkakati follows:] 
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1 GAO 
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Highlights 

Highlights of GAG-G8-425T, a statement 
before the Task Force for the Contested 
Election in the 13th Congressional District 
of Florida, Committee on House 
Administration. House of Representatives 


Why GAO Did This Study 

In November 2006, about 18,000 
undervotes were reported in 
Sarasota County in the race for 
Florida’s 13th Congressional 
District (Florida- 13). After the 
election results were contested in 
the House of Representatives, the 
task force unanimously voted to 
seek GAO's assistance in 
determining whether the voting 
systems contributed to the large 
undervote in Sarasota County. In 
October 2007, GAO presented its 
findings on the review of the voting 
systems and concluded that while 
prior tests and reviews provided 
some assurance that the voting 
systems performed correctly, they 
were not enough to provide 
reasonable assurance that the 
voting systems in Sarasota County 
did not contribute to the undervote. 
GAO proposed that a firmware 
verification test, a ballot test, and a 
calibration test be conducted. The 
task force requested that GAO 
proceed with the proposed 
additional tests. GAO also verified 
whether source code escrowed by 
Florida could be rebuilt into the 
firmware used in Sarasota County. 

To conduct its work, GAO 
conducted tests on a sample of 
voting systems used in Sarasota 
County during the 2006 general 
election. GAO witnessed the 
rebuild of the firmware from the 
escrowed source code at the 
manufacturer’s development 
facility. GAO reviewed test 
documentation from Florida, 
Saraiota County, and the voting 
system manufacturer and met with 
election officials to prepare the test 
protocols and detailed test 
procedures. 

To view the full product, including the scope 
and methodology, click on GAO-08-425T. 
For more information, contact Nabajyoti 
Barkakati at (202) 512-6412 or 
barkakatin@gao.gov. 


February 8. 2008 


ELECTIONS 

Results of GAO's Testing of Voting Systems Used in- 
Sarasota County in Florida's 13th Congressional 
District 


What GAO Found 

GAO conducted three tests on the iVotronic Direct Recording Electronic 
(DRE) voting systems in Sarasota County and these tests did not identify any 
problems. Based on its testing, GAO obtained increased assurance that the 
iVotronic DREs used in Sarasota County during the 2006 general election did 
not contribute to the large undervote in the Florida- 13 contest Although the 
test results cannot be used to provide absolute assurance, GAO believes that 
these test results, combined with the other reviews that have been conducted 
by the State of Florida, GAO, and others, have significantly reduced the 
possibility that the iVotronic DREs were the cause of the undervote. 

GAO’s firmware verification test showed that the firmware installed in a 
statistically selected sample of 1 15 machines used by Sarasota County during 
the 2006 general election matched the firmware certified by t he Florida 
Division of Elections. The statistical approach used in selecting these 
machines lets GAO estimate with a 99 percent confidence level that no more 
than 60 of the 1,499 iVotronic DREs that recorded votes in the 2006 general 
election were using different firmware. Consequently. GAO is able to place 
more confidence in the results of other tests conducted on a small number 
machines by GAO and by others, which indicated that the iVotronic DREs 
not cause the undervote. GAO also confirmed that when the manufacturer 
rebuilt the iVotronic DRE firmware from the source code that was held in 
escrow by the Florida Division of Elections and previously reviewed by GAO 
and others, the resulting firmware matched the version certified by the Florida 
Division of Electioas. 

For the ballot test, GAO cast predefined test ballots on 10 iVotronic DREs and 
confirmed that each ballot was displayed and recorded accurately. GAO 
conducted the calibration test by miscalibrating two iVotronic DREs and 
casting ballots on them to validate that the machines recorded the information 
that was displayed on the touch screen. Based on the results of the ballot and 
calibration tests, GAO found that (1) the machines properly displayed, 
recorded, and counted the selections for all test ballots cast during ballot 
testing involving 1 12 common ways a voter may have interacted with the 
system, and (2) the deliberately miscalibrated machines, though difficult to 
use, accurately recorded the ballot selections as displayed on screen. 

At this point, GAO believes that adequate testing has been performed on the 
voting machine software and does not r ecommend further testing in this area. 
Given the complex interaction of people, processes, and technology that must 
work effectively together to achieve a successful election, GAO acknowledges 
the possibility that the large undervote in Florida’s 13th Congressional District 
race could have been caused by factors such as voters who intentionally 
undervoted, or voters who did not properly cast their ballots on the iVotronic 
DRE, potentially because of issues relating to interaction between voters ami 
the ballot. 


.United States Government Accountability Office 
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Mr. Chairman and Members of the Task Force: 

I am pleased to appear before the task force today to present the findings 
on our testing of the voting equipment used in the 2006 general election in 
Florida’s 13th Congressional District (Florida-13). I would like to thank the 
task force for its overall support of our efforts and specifically for the 
assistance provided in obtaining resources from the House Recording 
Studio that were critical to successfully completing our testing efforts. 

In November 2006, about 18,000 undervotes were reported in Sarasota 
County in the race for Florida’s 13th Congressional District. 1 After the 
election results were contested in the House of Representatives, the task 
force met and unanimously voted to seek GAO’s assistance in determining 
whether the voting systems contributed to the large undervote in Sarasota 
County. In our October 2, 2007, statement for the task force, we presented 
the findings of our review of the voting systems and stated that while prior 
tests and reviews provided some level of assurance that the voting systems 
in Sarasota County — iVotronic direct recording electronic (DRE) voting 
systems manufactured by Election Systems and Software (ES&S) — 
functioned correctly, they were not enough to provide reasonable 
assurance that the iVotronic DRE voting systems did not contribute to the 
undervote. 2 Specifically, we found that assurance was lacking in three 
areas and proposed to the task force that additional tests — firmware 
verification, ballot, and calibration — be conducted to address these areas. 
We stated that successful accomplishment of these tests would provide 
increased, but not absolute, assurance that the iVotronic DREs used in 
Sarasota County during the 2006 general election did not cause the 
undervote. The task force requested that we proceed with the proposed 
additional tests. Our objectives were to (1) verify that firmware installed in 
a statistical sample of iVotronic DREs was identical to the firmware 
certified by the State of Florida, (2) perform ballot testing using 112 ways 
to cast a ballot for the Florida- 13 contest to ensure that the voting 
machines would properly record and count the ballots, and (3) 
deliberately misealibrate voting machines and then cast ballots on those 


1 Undervotes occur when the number of choices selected by the voter is fewer than the 
maximum allowed for that contest. In this case, it means ballots that, did not record a 
selection for either candidate in the congressional contest. 

2 GAO, Elections: Further Testing Could Provide Increased but Not Absolute Assurance 
That Voting Systems Did Nut Cause Undervotes in Florida’s 1 3th Congressional District, 
GAO-08-97T (Washington, D.C.: Oct. 2, 2007). 
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machines to ensure that the voting machines would properly record the 
ballots. As part of the first objective, w r e also validated that the source 
code, which was held in escrow by the Florida Division of Elections, 
would produce the firmware used by Sarasota County during the 2006 
general election. 

To conduct our tests, we developed test protocols and detailed test 
procedures. We met with officials from the Sarasota County Supervisor of 
Elections, the Florida Department of State and Division of Elections, and 
ES&S to obtain the necessary details about the voting systems and prior 
tests to document our test procedures. We also reviewed voting system 
documentation to develop a testing approach and the test procedures. To 
ensure that the certified firmware held in escrow by the Florida Division 
of Elections corresponded to the source code that was reviewed by a team 
from Florida State University and us, on November 19, 2007, we visited 
ES&S’s development facility in Rockford, Illinois, and witnessed the 
rebuild of the firmware from the escrowed source code. 

Further details on our test methodology are included in the following 
sections on each of the three tests. Appendix I outlines the process used i 
select machines for testing, and appendix II lists the iVotronic DREs that, 
we tested. We coordinated with the Florida Division of Elections and the 
Sarasota County Supervisor of Elections to obtain access to the iVotronic 
DREs and other necessary test equipment to conduct our testing. We 
conducted the firmware verification, ballot, and calibration tests at the 
Sarasota County Voting Equipment Facility (VEF) in Sarasota, Florida. We 
established the test environment on November 26, 2007, and conducted 
the tests from November 27, 2007, to December 4, 2007. During this time, 
we completed the steps necessary to conduct the tests and collected the 
test data. In addition, we video recorded the tests. One camera was used 
to capture a wide angle shot of the test room. Other cameras recorded the 
conduct of the firmware verification, ballot, and calibration tests. 

We provided a draft of this statement to the Florida Department of State 
and ES&S for their review and comments. We briefed the Sarasota County 
Supervisor of Elections on the contents of our statement. The Florida 
Department of State and ES&S also conducted a sensitivity review to 
ensure that business proprietary information is not disclosed in this 
statement. We conducted our work from October 2007 to February 2008 in 
Washington, D.C.; Tallahassee and Sarasota, Florida; and at, ES&S facilities 
in Rockford, Illinois, and Omaha, Nebraska. 
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Results in Brief 


We conducted three tests on the iVotronic DRE voting systems used in 
Sarasota County and these tests did not identify any problems that would 
indicate that the machines were responsible for the undervote in the 
Florida-13 race in the 2006 general election. In our firmware verification 
test, we extracted the firmware from a random probability sample of 115 
iVotronic DREs out of the 1,499 iVotronic DREs used in Sarasota County’s 
2006 general election and found that each machine’s firmware matched the 
certified version of firmware held in escrow by the Florida Division of 
Elections. The statistical approach used in selecting these machines 
enables us to estimate with a 99 percent confidence level that at least 1,439 
of the 1,499 machines used the same firmware that was certified by the 
Florida Division of Elections. Consequently, we have more confidence in 
the results of other tests conducted on a small number of machines by 
GAO and by others, which indicated that the iVotronic DREs were not the 
cause of the undervote. We witnessed the rebuild of the iVotronic DRE’s 
firmware from the source code that was held in escrow by the Florida 
Division of Elections and that was previously reviewed by Florida State 
University and by us. At ES&S’s software development facility, we 
observed that rebuilding the firmware from the escrowed source code 
resulted in the same firmware that was certified and held in escrow by the 
Florida Division of Elections. This validation provides greater confidence 
in the results of prior source code reviews by Florida State University and 
us. 

For the ballot test, we cast predefined test ballots on 10 iVotronic DREs 
and confirmed that each ballot was displayed and recorded accurately. 

The test ballots represented 1 12 common ways a voter may have 
interacted with the iVotronic DRE to select a candidate in the Florida- 13 
race and cast a ballot. These tests were performed on nine machines 
configured as election day machines and then repeated on one machine 
configured as an early voting machine. 

Finally, we conducted the calibration test by miscalibrating two iVotronic 
DREs and casting ballots on them to validate that the machines recorded 
the information that was displayed on the touch screen. Our tests, 
involving a total of 10 different miscalibration patterns and capturing 39 
ballots, found that the machines correctly displayed the selection in the 
Florida-13 race on the review screen and correctly recorded the ballot. 
Although the machines were more difficult to use, the selections shown on 
the screen were the same selections captured by the machine when the 
ballot was cast. 
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Based on the results of these tests, we have obtained increased assurance, 
but not absolute assurance that, the iVotronic DREs used in Sarasota 
County’s 2006 general election did not contribute to the large undervote in 
the Florida-13 contest. Absolute assurance is impossible to achieve 
because we are unable to recreate the conditions of the election in which 
the undervote occurred. Although the test results cannot be used to 
provide absolute assurance, we believe that these test results, combined 
with the other reviews that have been conducted by the State of Florida, 
GAO, and others, have significantly reduced the possibility that the 
iVotronic DREs were the cause of the undervote. At this point, we believe 
that adequate testing has been performed on the voting machine software 
to reach this conclusion and do not recommend further testing in this area. 
Given the complex interaction of people, processes, and technology that 
must work effectively together to achieve a successful election, we 
acknowledge the possibility that the large undervote in Florida’s 13th 
Congressional District race could have been caused by factors such as 
voters who intentionally undervoted, or voters who did not properly cast 
their ballots on the iVotronic DRE, potentially because of issues relating to 
interaction between voters and the ballot. 


Background 


The 13th Congressional District of Florida comprises DeSoto, Hardee, 
Sarasota, and parts of Charlotte and Manatee Counties. In the November 
2006 general election, there were two candidates in the race to represent 
the 13th Congressional District: Vem Buchanan, the Republican candidate, 
and Christine Jennings, the Democratic candidate. The State of Florida 
certified Vem Buchanan the winner of the election. The margin of victory 
was 369 votes out of a total of 238,249 votes counted. Table 1 summarizes 
the results of the election and shows that the results from Sarasota County 
exhibited a significantly higher undervote rate than in the other counties in 
the congressional district. 
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Table 1: Results from 2006 General Election for Florida Congressional District 13 


County 

Buchanan 

Jennings 

Undervotes 

Total 
ballots cast 

Percentage 

undervote 

Charlotte 

4.460 

4,277 

225 

8,962 

2.51 

DeSoto 

3,471 

3.058 

142 

6,672 

2.13 

Hardee 

2,629 

1,686 

269 

4,584 

5.87 

Manatee 

50,117 

44.432 

2,274 

96,828 

2.35 

Sarasota 

58,632 

65.487 

18,412 

142,532 

12.92 

Total 

119,309 

118,940 

21,322 

259,578 



Source: GAO analysis ol Florida Division of Elections. Cnarfotte County, DeSoto County. Haidee County. Manatee County, and 
Sarasota County data. 

Note: Numbers do not add up because of overvotes-where voters select more than the maximum 
number of candidates allowed in a race; in this case, an overvote was a ballot that had votes lor both 
Buchanan and Jennings. 

As seen in table 1, about 18,000 undervotes were reported in Sarasota 
County in the race for Florida’s 13th Congressional District. After the 
election results were contested in the House of Representatives, the task 
force met and unanimously voted to seek GAO’s assistance in determining 
whether the voting systems contributed to the large undervote in Sarasota 
County. On June 14, 2007, we met with the task force and agreed upon an 
engagement plan. We reported on the status of our review at an interim 
meeting held by the task force on August 3, 2007. 3 

On October 2, 2007, we reported that our analysis of election data did not 
identify any particular voting machines or machine characteristics that 
could have caused the large undervote in the Florida-13 race. 4 The 
undervotes in Sarasota County were generally distributed across all 
machines and precincts. We found that some of the prior tests and reviews 
conducted by the State of Florida and Sarasota County provided assurance 
that certain components of the voting system in Sarasota County 
functioned correctly, but they were not enough to provide reasonable 
assurance that the iVotronic DREs did not contribute to the undervote. We 
proposed three tests — firmware verification, ballot, and calibration — to 
provide increased assurance, but not absolute assurance, that the 
iVotronic DREs did not cause the large undervote in Sarasota County. We 


'* GAO, Elections: Status of GAO's Review of Voting Equipment Used in Florida's 13th 
Congressional District-, GAO-07-1 167T (Washington, D.C.: Aug. 3, 2007). 

4 GAO-OS-97T. 
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stated that the successful conduct of the tests could reduce the possibility 
that the voting systems caused the undervote and shift attention to the 
possibilities that voters intentionally undervoted or voters did not properly 
cast their ballots on the iVotronic DRE, potentially because of issues 
relating to interaction between voters and the ballot. 


Overview of the Voting In the 2006 general election, Sarasota County used voting systems 

Systems Used in Sarasota manufactured by ES&S. The State of Florida has certified different 
Cm i n tv in tho 900f t Cnnornl versions of ES&S voting systems. The version used in Sarasota County was 
Elections designated ES&S Voting System Release 4.5, Version 2, Revision 2, and 

consisted of iVotronic DREs, a Model 650 central count optical scan 
tabulator for absentee ballots, and the Unity election management system. 
It was certified by the State of Florida on July 17, 2006. The certified 
system includes different configurations and optional elements, several of 
which were not used in Sarasota County. 5 

The election management part of the voting system is called Unity; the 
version that was used was 2.44.2. Figure 1 shows the overall election 
operation using the Unity election management system and the iVotronic 
DRE. 


in May 2007, the State of Florida enacted legislation requiring, in general, the use of 
optical scan voting equipment that provides a paper trail. These requirements are effective 
July I. 2008. There is an exemption from these requirements for voting by persons with 
disabilities. 
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Figure 1 : Overview of Election Operation Using the Unity Election Management System and iVotronic DRE 
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Sarasota County used iVotronic DREs for early and election day voting. 
Specifically, Sarasota County used the 12-inch iVotronic DRE, hardware 
version 1.1 with firmware version 8.0. 1.2.’ Some of the iVotronic DREs are 
configured to use audio ballots, which are often referred to as Americans 
with Disabilities Act (ADA) machines. The iVotronic DRE uses a touch 
screen — a pressure-sensitive graphics display panel — to display and 
record votes (see fig. 2). 


h The certified version ofES&S Voting System Release 4.5, Version 2, Revision 2, specifies 
the use of iVotronic hardware version 1.0. According to Florida Division of Election 
officials, hardware version i. 1 of the iVotronic DRE has been available since at least 2004 
and should have been included as apart of the certification for BS&S Voting System 
Release 4.5, Version 2, Revision 2, According to ES&S officials, iVotronic firmware version 
8.0. 1.2 rims in exactly the same manner on hardware versions 1.0 and L I. 
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Figure 2: The iVotronic DRE Voting System anti Sts Components 



The machine has a storage case t hat also serves as the voting booth. The 
operation of the iVotronic DRE requires the use of a personalized 
electronic ballot (PER), which is a storage device with an infrared window 
used for transmission of ballot data to and from the iVotronic DRE. The 
iVotronic DRE has four independent flash memory modules, one of which 
contains the program code — firmware — that runs the machine; the 
remaining three flash memory modules store redundant copies of ballot 
definitions, machine configuration information, ballots cast by voter's, and 
event logs (see fig. 3). The iVotronic DRE includes a VOTE button that the 
voter has to press to cast a ballot and record the information in the flash 
memory. The iVotronic DRE also includes a compact flash card that can 
be used to load sound files onto iVotronic DREs with ADA functionality. 
The iVotronic DRE’s firmware can be updated through the compact flash 
card. Additionally, at the end of polling, the ballots and audit information 
are to be copied from the internal flash memory module to the compact 
flash card. 
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Figure 3: inside View of the iVotronic DRE Showing the Flash Memory Modules 


Source: GAO. 
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To use the iVotronic DRE for voting, a poll worker activates the iVotronic 
DRE by inserting a FEB into the PEB slot after the voter has signed in at 
the polling place. After the poll worker makes selections so t hat the 
appropriate ballot will appear, the PEB is removed and the voter is ready 
to begin using the system. The ballot is presented to the voter in a series of 
display screens, with candidate information on the left side of the screen 
and selection boxes on the right side (see fig. 4). 
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Figure 4: Second Ballot Page Showing the Congressional and Gubernatorial Races 
in Sarasota County's 2006 Genera! Election 
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Source: Saraaela County Supervisor at Elections. 


The voter can make a selection by touching anywhere on the line, and the 
iVotronic DRE responds by highlighting the entire line and displaying an X 
in the box next to the candidate’s name. The voter can also change his or 
her selection by touching the line corresponding to another candidate or 
by deselecting his or her choice. “Previous Page” and “Next Page” buttons 
are used to navigate the multipage ballot. After completing all selections, 
the voter is presented with a summary screen with all of his or her 
selections (see fig, 5). From the summary screen, the voter can change any 
selection by selecting the race. The race will be displayed to the voter on 
its own ballot page. When the voter is satisfied with the selections and has 
reached the final summary screen, the red VOTE button is illuminated, 
indicating the voter can now cast his or her ballot. When the VOTE button 
is pressed, the voting session is complete and the ballot is recorded on the 
iVotronic DRE. In Sarasota County’s 2006 general election, there were nine 
different ballot styles with between 28 and 40 races, which required 
between 15 and 21 electronic ballot pages to display, and 3 to 4 summary 
pages for review purposes. 
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Figure 5: First Summary Page in Sarasota County's 2006 Genera? Election 
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Source: Sarasota County Supervisor of Elections. 


Election Systems Involve 
People, Processes, and 
Technology 


An election system is based upon a complex interaction of people (vot ers, 
election officials, and poll workers), processes (controls), and technology 
that must work effectively together to achieve a successful election. The 
particular technology used to cast and count votes is a critical part of how 
elections are conducted, but it is only one facet of a multifaceted election 
process that involves the interplay of people, processes, and technology. 


As we have previously reported, every stage of the election process — 
registration, absentee and early voting, preparing for and conducting 
Election Day activities, provisional voting, and vote counting — is affected 
by the interaction of people, processes, and technology. 7 Breakdowns in 
the interaction of people, processes, and technology may, at any stage of 


' GAO, Elections: The Nation's Evolving Election System as Reflected in the November 
2004 General Election, GAO-06-450 (Washington, D.C.: June (5, 2006). 
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aii election, impair an accurate vote count. For example, if the voter 
registration process is flawed, ineligible voters may be allowed to cast 
votes. Poll worker training deficiencies may contribute to discrepancies in 
the number of votes credited and cast, if voter information was not 
entered properly into poll books. Mistakes in using the DRE systems could 
result from inadequate understanding of the equipment on the part of 
those using it. 

As noted in our October statement, we recognize that human interaction 
with the ballot layout could be a potential cause of the undervote, and we 
noted that several suggestions have been offered as possible ways to 
establish that voters are intentionally undervoting and to provide some 
assurance that the voting systems did not cause the undervote.* For 
instance, 

• A voter-verified paper trail could provide an independent confirmation 
that the touch screen voting systems did not malfunction in recording 
and counting the votes from the election. The paper trail would reflect 
the voter’s selections and, if necessary, could be used in the counting 
or recounting of votes. This issue was also recognized in the source 
code review performed by the Security and Assurance in Information 
Technology (SAIT) laboratory at Florida State University as well as the 
2005 and draft 2007 Voluntary Voting Systems Guidelines prepared for 
the Election Assistance Commission. We have previously reported on 
the need to implement such a function properly. 9 

• Explicit feedback to voters that a race has been undervoted and a 
prompt for voters to affirm their intent to undervote might help prevent, 
many voters from unintentionally not casting a vote in a race. On the 
iVotronic DREs, such feedback and prompts are provided only when 
the voter attempts to cast a completely blank ballot, but not. when a 
voter fails to vote in individual races. 

• Offering a “none of the above” option in a race would provide voters 
with the opportunity to indicate that they are intentionally undervoting. 
For example, the State of Nevada provides this option in certain races 
in its elections. 


* gao-os-ptt. 

9 GAO, Elections: Federal Efforts to Improve Security and Reliability of Electronic Voting 
Systems Are Under Way. but Key Activities Need to Be Completed, GAO-Of>-O3f> 
(Washington, D.C.: Sept. 21, 2005). 
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We reported that decisions about these or other suggestions about ballot 
layout or voting system functions should be informed by human factors 
studies that assess such measures' effectiveness in accurately recording 
voters’ preferences, making voting systems easier to use, and preventing 
unintentional undervotes. 


Tests Confirm 
Sarasota County 
iVotronic DREs Used 
Same Firmware 
Certified by Florida 


We previously reported that having reasonable assurance that all iVotronic 
DREs that recorded votes in the 2006 general election were running the 
same certified firmware would allow us to have more confidence that the 
iVotronic DREs will behave similarly when tested.' 0 Consequently, if we 
are reasonably confident that the same firmware was running in all 1,499 
machines, then we are more confident that the results of other tests, 
conducted both by GAO and by others, on a small number of machines can 
be used to obtain increased assurance that the iVotronic DREs did not 
cause the undervote. We also reported that there was a lack of assurance 
that the source code that was held in escrow by the Florida Division of 
Elections and that was previously reviewed by Florida State University 
and by us, if rebuilt, would corresponded to the firmware that was 
certified and held in escrow by the Florida Division of Elections, We found 
that the firmware on a statistically selected sample of 115 iVotronic DREs 
was the same as that certified by the Florida Division of Elections. We also 
found that the escrowed source code, when rebuilt into executable 
firmware, corresponded to the 8.0. 1.2 firmware that was certified by the 
Florida Division of Elections. 


Methodology for Firmware Our methodology to obtain reasonable assurance that the firmware used 
Verification Testing on Sarasota County’s iVotronic DREs during the 2006 general election was 

the same as that certified by the State of Florida was broken down into 
two basic steps: (1) selecting a representative sample of machines, and (2) 
verifying that the firmware extracted from the voting machines was the 
same as the escrowed firmware that had been certified by the Florida 
Division of Elections. Appendix I details the methodology for selecting the 
representative sample of machines. Appendix II contains a list of the serial 
numbers of the tested iVotronic DREs. 

To ensure that we would be testing with the iVotronic firmware certified 
by the Florida Division of Elections, on October 18, 2007, we and officials 


19 GAO-OS-97T. 
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from the Florida Division of Elections made two copies of the escrowed 
iVotronic 8.0. 1.2 firmware on compact discs (CD) and placed them in two 
tamper-evident bags with serial numbers. The bags were subsequently 
hand-delivered by a Florida Division of Elections official for our use In the 
firmware verification test and for the rebuilding of the firmware from the 
source code. 

In order to extract the firmware from an iVotronic DRE, the machine was 
placed on an anti-static mat and the case was opened using a special 
screwdriver. After lifting the case, a special extraction tool was used to 
remove the flash memory module that contains the firmware. The flash 
memory module was then inserted in the socket of a Needham 
Electronics’ EMP-300 device that was connected to the universal serial bus 
(USB) port of a personal computer (PC). The EMPWin application running 
on that PC was used to read the firmware from the flash memory module 
and save the extracted firmware on the PC. The Florida Division of 
Elections loaned us the EMP-300 and EMPWin application for use in 
extracting firmware from the flash memory module. 

To compare the extracted firmware with the escrowed version, we relied 
on two commercially available software programs. First, we acquired a 
license for PrestoSoft’s ExamDifT Pro software that enables comparison of 
files. The ExamDiff Pro software is a commercially available program 
designed to highlight the differences between two files. For each selected 
iVotronic DRE, the extracted firmware was compared with the escrowed 
version with any differences highlighted by the program. 

Second, to further ensure that the extracted firmware matched the 
escrowed firmware, we compared the SHA-1 hash value of the extracted 
firmware to the hash value of the comparable certified firmware." We 
computed the SHA-1 hash by using the Maresware hash softw e that was 
provided by the Florida Division of Elections. In order to ensure that the 
commercial Maresware hash software properly calculated the SHA-1 hash 
value, we (1) created four files and obtained a fifth file that contained 


! 1 The National Institute of Standards and Technology (NIST) has issued a Federal 
Information Processing Standard (FIPS) chat describes four hashing algorithms that are 
iterative, one-way hash functions that can process a file and produce a condensed 
representation called a message digest or “hash." These algorithms enable the user to 
validate a file’s integrity since any change to the file will, with a very high probability, result 
in a different message digest. The technical details of this process are contained in FIPS 
180-2. 'Hie algorithm selected lor this testing effort is commonly referred to as SHA-1 and is 
the same algorithm used by the Florida Division of Flections during its audit. 
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executable code, (2) obtained hash values for each file by either using an 
external program that generated the hash values using the same hashing 
algorithm as the commercial product or using known hash values, 12 and (3) 
used the commercial program acquired for testing the firmware to ensure 
that the hash values it generated for these five files were identical to the 
expected hash values for those files. In each case, the hash values 
generated by the commercial program were identical to the expected 
values. Accordingly, reasonable assurance for the purposes of our review 
was obtained that the commercial program produced its hash values in 
accordance with the NIST algorithm. 

At the end of each day, we (1) used the commercial Maresware software to 
compute hash values for each of the firmware programs that had been 
unloaded during that day and all previous days, and (2) compared each 
hash created by this program to the expected value that was calculated 
from the firmware that had been escrowed by the Florida Division of 
Elections. This comparison provided further assurance that the extracted 
firmware was (1) identical to the version escrowed by the Florida Division 
of Elections when the hashes agreed, or (2) different if the hashes did not 
agree. 

We also verified that sequestered machines were not used since the 2006 
general election. For each of these sequestered machines, we used an 
audit PEB to copy the audit logs onto a compact flash card and then used 
the Unity election reporting manager to generate event log reports. We 
examined the event logs for the date and time of occurrence of activities 
that would indicate whether the machine had been used. Lack of such 
activities since the 2006 general election provided reasonable assurance 
that the machines had not been used since they were sequestered. 12 

In addition, to verify that the source code for iVotronic DRE firmware 
version 8.0.1.2 previously examined by the Florida State University SAIT 
source code review team and by GAO corresponded with the version 


12 Two of the files and the expected values used came from FIPS 180-2. 

111 We verified that, sequestered machines were not. used since the 2006 general election by 
(1) verifying that the seals placed on these machines agreed with Sarasota County’s 
records, and (2) checking the event logs maintained on the machine to determine whether 
the machines had been used since the machine had been sequestered. In every' case, we 
found that, the seal numbers agreed with Sarasota County's records. We were able to check 
the event log for 57 of the 58 sequestered iVotronic DREs. We were unable to power up the 
remaining iVotronic DRE and were consequently unable to extract tire needed audit data. 
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certified by the Florida Division of Elections, ES&S officials stated that it 
still had the development environment that could be used to compile, or 
rebuild, the certified firmware from the source code retained in escrow by 
the Florida Division of Elections. 1 1 As we previously noted, a software 
review and security analysis of the iVotronic DRE firmware was 
conducted by a team led by Florida State University’s SAIT laboratory/ 1 
The software review team attempted to confirm or refute many different 
hypotheses that, if true, might explain the undervote in the race for the 
13th Congressional District. In doing so, they made several observations 
about the source code, which we were able to independently verily. 

The rebuilding of the firmware was conducted by ES&S at its Rockford, 
Illinois, facility on November 19, 2007, and witnessed by us. Prior to the 
rebuild, the Florida Division of Elections provided an unofficial copy of 
the source code to ES&S so that ES&S could prepare the development, 
environment and test the rebuild steps. Using the official sealed copy of 
the source code CD, ES&S rebuilt the firmware in front of GAO 
representatives. ES&S described the development environment and we 
inspected it to satisfy ourselves that the firmware was faithfully rebuilt 
using the escrowed source code. After the rebuilding of the firmware, tht 
certified version of 8.0. 1.2 firmware was compared with the rebuilt version 
using PrestoSoft’s ExamDiff Pro. 


Results of Firmware While the Florida audit team had previously confirmed that the firmware 

Verification Testing running on six iVotronic DREs matched the certified version held in 

escrow by the Florida Division of Elections, we found that the sample size 
was too small to support generalization to all 1,499 iVotronic DREs that; 
recorded votes during the 2006 general election. Accordingly, we 
conducted a firmware verification test on a statistically valid sample ofl 15 
iVotronic DRE machines used by Sarasota County during the 2006 general 
election. The selected machines fell into two groups — machines that had 
not been used since the 2006 general election (referred to as sequestered 


“ la our October 2007 statement, we reported that according to ES&S, firmware compiled 
from the Florida escrowed source code may not be exactly identical to the firmware 
certified by the Florida Division of Elections because the embedded date and time stamp in 
the firmware would be different We found that the date and time was not embedded in the 
firmware and that an identical version could be created. 

1 ’ Security and Assurance in Information Technology Laboratory, Florida State University, 
Software Review and Security Analysis of the ES&S i Votronic 8.0. 1.2 Voting Machine 
Firmware (Tallahassee, Florida: Feb. 23, 2007). 
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machines) and machines that had been used in subsequent elections. For 
each machine, we extracted the firmware from a flash memory module in 
that machine and then compared the extracted firmware with the 
escrowed version using commercially available file comparison tools to 
determine whether they agreed. We found that the firmware installed in 
the flash memory module of each machine matched the escrowed 
firmware that had been certified by Florida. The statistical approach used 
to select these machines lets us estimate with a 99 percent confidence 
level that at least 1,439, or 96 percent, of the 1,499 machines used in the 
2006 genera] election used the firmware that was certified by the State of 
Florida. 

We witnessed the rebuild of the iVotronic DRE’s firmware from the source 
code that was held in escrow by the Florida Division of Elections and that 
was previously reviewed by Florida State University and by us. At ES&S’s 
software development facility, we observed that rebuilding the firmware 
from the escrowed source code resulted in the same firmware that was 
certified and held in escrow by the Florida Division of Elections. The 
comparison of the escrowed firmware to the version that was rebuilt by 
the vendor identified no differences and provides us reasonable assurance 
that the escrowed firmware corresponded to the escrowed source code. 
The successful rebuilding of the firmware from the escrowed source code 
enables us to have greater confidence in the conclusions derived from 
prior source code reviews by Florida State University and us. 


Ballot Testing Showed 
That Machines 
Accurately Recorded 
and Counted Ballots 


In our October 2007 statement, we noted that there were 1 12 common 
ways a voter may interact with the system to select a candidate in the 
Florida- 13 race and cast the ballot, and that prior testing of the iVotronic 
DREs covered only 13 of these 112 possible ways. We developed 224 test 
ballots to verify that the iVotronic DRE could accurately capture ballots 
using each of these 1 12 common ways a voter may interact with the 
system; 1 12 test ballots were cast on one machine configured for early 
voting, and another 1 12 ballots were cast on nine machines configured for 
election day voting. Our tests showed that for each of the 224 test ballots, 
the iVotronic DRE correctly captured each vote as cast for the Florida-13 
race. We also conducted firmware verification tests on these machines and 
verified that they were running the certified firmware. 


Methodology for Ballot The methodology for ballot testing can be broken into two major areas — 
Testing development, of the test ballots and execution of the test using those 

ballots. The following sections discuss these areas. 
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Development of Test Ballots In examining how the system allowed voters to make a selection in the 
Florida-13 race, we found at least 1 12 different ways a voter could make 
his or her selection and cast the ballot in the Florida-13 race, assuming 
that it was the only race on the ballot. Specifically, a voter could (1) 
initially select either candidate or neither candidate (i.e., undervote), (2) 
change the vote on the initial screen, and (3) use a combination of features 
to change or verify his or her selection by using the page back and review 
screen options. Accordingly, we tested these 1 12 ways to select a 
candidate on the early voting machine and on the election day machines 
(224 test ballots in total). 

The 112 standard test ballots cover all combinations of the following types 
of voter behavior: 

• Voter makes selection on the initial ballot screen and makes no 
changes or takes any other action to return to the contest to review or 
change selection. 

• Voter makes selection on the initial ballot screen and decides before 
leaving that screen to change the selection because of an error in 
selecting the candidate or for some other reason. 

• Voter makes selection on the initial ballot screen and then decides to 
use the page back option to review or change selection. 

• Voter makes selection on the initial ballot screen and continues to the 
review screen and then decides to use the review screen option to 
review or change selection. 

• Voter makes selection on the initial ballot screen and uses a 
combination of page back and review screen options to review or 
change selection. 

In each instance where a selection could be made, three choices were 
possible for the Florida-13 race: a selection for one of the two candidates, 
or no selection (i.e., an undervote )- 

In developing the standard test ballots, we did not consider all 
combinations of some other types of voter behavior that would have 
significantly increased the number of test cases wit hout providing 
significant benefits. In most cases, such behavior are variants of the 
primary voter behavior that we examined. The following are examples of 
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voter behavior that were not included in the standard test set in order to 
reduce the number of test cases to practicable levels: 

• Using a one-touch or two-touch method to make changes on a ballot 
page. 16 

• Varying the number of pages a voter may go back (“page backs”) to 
return to the page containing the Florida- 13 race to change or review a 
selection. 

• Casting a ballot from the review screen selection. The VOTE button is 
not activated until the voter reaches the last review screen. However, 
once the VOTE button has been activated, a ballot may be cast from 
any screen. For example, a voter may activate the VOTE button and 
then return to a contest to review or change the selection using the 
review screen option. Once the voter goes to the contest from the 
review screen and makes any desired changes, the voter can then cast 
the ballot from that screen rather than going back to the last page of 
the review screen or even the review screen that was used to return to 
the selection. 

Although we did not consider all combinations of these types of voter 
behavior when developing the standard test ballots, we included some of 
these user interactions in the execution of applicable test ballots to 
provide increased assurance that the system would handle these voter 
behaviors. For each applicable test ballot, we randomly determined the 
test procedure that, should be used for the following attributes: 

• Initial change method - The standard test ballots address voters 
making changes on the initial ballot screen. Where possible, the 
method used to change (one-touch or two-touch) the selection was 
randomly selected. 


1,1 The iVotronic DREs used in Sarasota County allow the user to make changes using two 
methods. The first method allows the user to simply touch the other candidate; e g., 
Candidate A is initially selected and the voter decides to select Candidate B by touching the 
name of Candidate B. We referred to this as the "one-touch method." The other method, 
referred to as the “two-touch method," involves the user first deselecting the initial choice 
and then making another selection; e.g., Candidate A is initially selected and the voter 
decides to select Candidate B by (l) touching the name of Candidate A, which deselects 
Candidate A, and then (2) touching the name of Candidate B to select it. 
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• Number of page backs - The ballots used by Sarasota County 
included the page back function. After reviewing the ballots, it 
appeared reasonable to expect that voters who may have used the page 
back option would probably decide that they had missed the race by 
the time they went one or two pages beyond the page with the 
Florida- 13 race. Therefore, when a standard test ballot contained a 
page back requirement, the number of page backs was randomly 
selected to determine whether one or two page backs should be used. 

• Page back change method - Some test ballots required a change after 
the page back option was selected. As with the initial change method, 
where possible, the method of changing (one-touch or two-touch) the 
selection was randomly assigned. 

• Review screen change method - The system displays a review 
screen that shows the voter’s selections (or lack of selections) after the 
voter has progressed through all contests. On the review screen, the 
voter can select, a race to go directly to that contest and (1) review the 
selection made, and (2) make any desired corrections. The standard 
test ballots were designed to cover this type of event. Where possible, 
the method used to make the change (one-touch or two-touch) was 
randomly selected. 

• Activate VOTE button and cast ballots from the review screen - 

In order to test casting ballots from locations other than the last review 
screen, the VOTE button must be activated prior to going to a screen 
where the ballot is cast. 17 In order to determine which test ballots 
should be used for this test, a two-step approach was adopted. First, a 
random selection of the ballots that use the review screen option was 
made to determine which test ballots should have the VOTE button 
activated. Then a random selection of t hese test ballots was made to 
determine whether the ballot should be cast from the review screen 
selection. 

Besides those attributes that directly affect the selection in the Florida-13 

race, we varied the other attributes on the ballot in order to complete the 


*' The actual procedure Is to (1) go to the last review screen, which activates the VOTE 
button, (2) page back to the contest (normally 2 or 3 page backs depending on the ballot: 
style), and (3) selecting the contest on the review screen that should be revisited. We 
assumed that voters would cast such ballots using this procedure instead of using the page 
back option because it did not appeal" reasonable that a voter would page back at least 17 
screens to reach die Florida-13 race, which was the focus of the testing. 
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ballot test. For each of the 224 test ballots, we used random values for 
other attributes, including the following: 

• Ballot style - Each ballot was randomly assigned one of the nine 
ballot styles used in the election. 

• Write-in candidate - All ballot styles includes write-in options in at 
least 2 races — United States Senate and State Govemor/Lieutenant 
Governor. To verily that the iVotronic DRE accurately recorded the 
selection in the Florida-13 race for each test ballot, we needed a way to 
identify each test ballot in the ballot image log. To accomplish this, we 
randomly selected one of these two races, selected the write-in 
candidate for the race, and entered a unique value (i.e., the test ballot 
number) in the write-in field. 

• Candidates and selections in other races on the ballot - Each 
ballot style had between 28 and 40 contests on the ballot. The values 
for the contests besides the Florida- 13 race and the write-in field were 
also randomly selected. For example, most items had three possible 
choices — candidate 1 (or Yes), candidate 2 (or No), and undervote. 
Which of these three values was used for a given contest was randomly 
determined. 

The values used for these attributes were independently determined for 
the election day and early voting test ballots. For example, Test Ballot 2 
(election day) and Test Ballot 202 (early voting) were designed to test the 
same standard condition described by one of the 112 standard test 
ballots. ls Table 2 illustrates some of the similarities and differences 
between the two test ballots that result from the random selection process 
used to determine the other aspects of the ballot. 


!b The standard actions taken in these two test ballots called for the tester to (1) make a 
selection on the initial screen and then change the selection, (2) page back to the initial 
selection screen and change the selection, and (3) use the review screen option to change 
the selection again. 


GAO-08-425T 



31 





Table 2: Examples of Differences between Test Ballot 2 and Test Ballot 202 


Test item 

Test Ballot 2 

Test Ballot 202 

Precinct 

142 

143 

Ballot style 

6 

7 

Contest used to contain unique value used to identify the test ballot 
during the review process. 

Govemor/Lieutenant 

Governor 

U.S. Senate 

Method used to make change on initial screen for contest 

Two-touch 

One-touch 

Number of page backs to return to contest 

2 

1 

Method used to make change after paging back to contest 

Two-touch 

Two-touch 

Activate VOTE button prior to using the review screen to return to the 
contest 

No 

Yes 

Selection for Attorney General 

McCollum 

Campbell 

Selection for Constitutional Amendment 1 

No 

Undervote 

Selection for Constitutional Amendment 8 

No 

No 

Method used to make change using the review screen approach 

Two- touch 

Two-touch 

Cast ballot from contest selection 

No 

Yes 

Return to review screen and then cast ballot 

Yes 

No 


Source: GAO 


Finally, we selected 10 random machines to be used for the ballot testing.' 3 
One machine was selected from those that were used in early voting in the 
2006 general election. The other nine were selected from those that used 
each of the ballot styles on election day in the 2006 general election. 3 " For 
each election day machine, the assigned precinct was the same as the 
precinct where the machine was used during the 2006 general election. 

For the early voting machine, we needed to assign precincts for each 
ballot style. We used the precinct associated with the back-up machine 
used for election day testing as the precinct for that ballot style. " If the 
first back-up machine was assigned the same precinct number as the 
primary election day machine, then we used the precinct associated with 
the second back-up machine. This approach was taken to maximize the 
number of precincts used in the testing efforts. 


u Details on the random selection can be found in apjtendix I. 

We excluded machines from one precinct that used two ballot styles instead of one. 

In order to ensure that we could complete our tests even if a machine selected fortesting 
failed to operate, our statistical sampling methodology generated a list of mad tines that 
could t>e used as replacements and still maintain the integrity of the testing process. These 
arc referred to as “back-up” machines. 


Page 22 


GAO-G8-425T 




32 


Process Used in Executing the A two-person test team conducted the ballot testing. One tester read out 
Ballot Test aloud the steps called for in the test ballot while the other tester 

performed those actions. In order to ensure that all of the actions relating 
to the Florida-13 congressional race were performed as laid out in the test 
ballots, a two-person review team observed a video display of the test and 
compared the actions taken by the tester to those called for in the test 
ballot. Furthermore, after the testing was completed, another team 
reviewed the video recording of these teste to validate that the actions 
relating to the Florida-13 contest taken by the tester were consistent with 
those called for by the test ballots.” 

The criteria used to determine whether the test produced the expected 
result was derived from the Florida Voting System Standards. 2 ' 1 
Specifically, among other things, these standards require the system to 
allow the voter to (1) determine whether the inputs given to the system 
have selected the candidates that he or she intended to select, (2) review 
the candidate selections made by the voter, and (3) change any selection 
previously made and confirm the new selection prior to the act of casting 
the ballot. Furthermore, the system must communicate to the voter the 
fact that the voter has failed to vote in a race (undervote) and require the 
voter to confirm his or her intent to undervote before casting the ballot. 
During the ballot test, the actual system response was compared to the 
expected results by a review team and after the testing was completed 
another review team compared the video records to the test ballots to 
validate that the tests had been performed in accordance with test scripts 
for the Florida- 13 contest. 

At the beginning of testing on each iVotronic DRE, the machine was 
opened for voting and a zero tape was printed. After the casting of all test 
ballots on the machine, the machine was closed and a results tape was 
printed. The closing of the machine also writes the audit data to the 
compact flash card, including event data and ballot images. We examined 
the results tapes and compared the total votes cast for the Florida-13 
contest against what was expected from the test ballots. We also kept 


22 These two reviews identified two early voting and seven election day test ballots where 
the specified scripts were not followed exactly for the Florida- 13 contest. Because these 
test ballots had not followed the test script for the Florida- 13 contest exactly, they were 
retested. Accordingly, the testing efforts resulted in 233 actual ballots being cast. 

21 Florida Department of State, Florida Voting System Standards, Form DS-DE 101 (Jan. 
12, 2005). 
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track of the total number of ballots handled by the machine, called the 
“protective count” of an iVotronic DRE, before and alter the test and 
confirmed that the increase in protective count matched the number of 
test ballots cast on that machine. 21 

Using the Unity election reporting manager, we read the compact flash 
cards and processed the audit data on each ballot test machine. We 
generated the ballot image log and examined the individual test ballots in 
the ballot image log. We looked for the unique identifier that was used for 
each test ballot and then confirmed that the ballot image reflected the 
correct selection for the Florida- 13 race as called for by the test ballot. For 
example, the test script for Test Ballot 1 required the tester to (1) select a 
write-in candidate for U.S. Senate and (2) enter the value of “TB1” in the 
write-in field. Because only this test ballot used this value, we could 
review the ballot image log to determine what selection the voting 
machine recorded for the Florida-13 contest for the ballot showing “TBl” 
as the write-in candidate for U.S. Senate. 25 

Finally, using the process discussed previously for firmware testing, the 
firmware on all machines used for ballot testing was validated to ensure 
these machines used the same firmware that had been certified by the 
Florida Division of Elections. 


Results of Ballot Testing After executing the ballot tests on the election day and early voting 

machines, we found that all 10 iVotronic DREs captured the votes for the 
Florida-13 race on the test ballots accurately. We used a unique identifier 
in a write-in field in each test ballot and verified that the iVotronic DRE 
accurately captured the tester’s final selections in the Florida- 13 race for 
each test, ballot. 


The iVotronic DRE is designed to maintain a count of all ballots cast on a given machine 
and functions much like an automobile's odometer. The protective count can be used to 
help ensure that the election process did not lose any votes. For example, before a machine 
is sent to a precinct, the protective count is recorded. Accordingly, if the precinct's voting 
register show that 100 individuals voted, then the increase in the protective counts for all 
machines assigned to that precinct should increase by 100. This value can then be 
compared to the actual votes recorded in the election to ensure that the values are 
consistent; i.e., the results tape for the election shows that. 100 voles have been accounted 
for during this election using this example precinct. 

25 In some cases, a test ballot had to be reentered because the original test did not follow all 
of the desired actions associated with the Florida- 13 contest. In these cases, the value 
entered was made unique by adding a letter to the value, e.g., “TBl A". 
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Testing 112 ways to select a candidate on a single machine also provided 
us some additional assurance that the volume of ballots cast on election 
day did not contribute to the undervote. We noted that casting 1 12 ballots 
on a single machine was more than the number of ballots cast on over 99 
percent of the 1,415 machines used on election day. 


Deliberately 
Miscalibrated 
iVotronic DREs 
Accurately Recorded 
Displayed Ballots 


Because little was known about the effect of a miscalibrated machine on 
the behavior of an iVotronic DRE, we deliberately miscalibrated two 
iVotronic DREs using 10 different miscalibration methods to verily the 
functioning of the machine. Although the miscalibration made the machine 
more difficult to use, the 39 ballots used in this test confirmed that the 
system correctly recorded the displayed vote for the Florida-13 contest 
and did not. appear to contribute to the undervote. 


Methodology for For the calibration testing, we judgment ally selected five different 

Calibration Testing miscalibration patterns and repeated each pattern twice — once with a 

small amount of miscalibration and the second time with a large amount of 
miscalibration. The amount of miscalibration was also subjective — 
roughly 0.25 to 0.5 inch for a small amount and about 0.7 to 1 inch for a 
large miscalibration. 

The miscalibration patterns are shown in the following figures. 
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39 


Figure 10: Miscalibration Pattern 5: For Each Calibration Point, the fester touches 

a Point Shifted Horizontally to the Left 



We conducted calibration testing on two different machines that were 
used for ballot, testing.® As with ballot testing, at the beginning of testing 
of each machine, we opened the machine for voting and printed a zero 
tape. During the opening process, we calibrated the machine with one of 
the miscalibration patterns. After the machine was miscalibrated, we then 
executed at least three of the test ballots that were used during ballot 
testing on that machine for each test . 27 The test ballots were rotated among 


26 The approach used to select these machines is described in appendix I. 

In the testing of the first two miscalibration patterns for the first machine, all. the test 
ballots used in the ballot, testing for that machine were repeated. However, the individual 
performing the testing sewn recognized the changes that were needed to compensate for 
the miscalibration. Accordingly, the tester did not make as many attempts to perform the 
desired function in the later cases as with the first three cases. Therefore, for the remaining 
eight miscalibration test patterns, we executed three test ballots per pattern because these 
cases produced the greatest likelihood of generating spurious touches before obtaining the 
desired selection. 
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the miscalibration patterns. For example, one of the machines had eight 
different ballot test scripts. The first three were used on one 
miscalibration pattern, the next three on another miscalibration pattern, 
and the final two plus the first one would be used on another 
miscalibration pattern. After the ballots were cast for one miscalibration 
pattern, the machine would be miscalibrated with another pattern. After 
the needed miscalibration patterns were tested on a machine, the 
iVotronic DRE was closed and a results tape was printed. The closing of 
the iVotronic DRE also wrote the audit data to the compact flash card. 

During the testing, the tester was instructed to take whatever actions were 
necessary to achieve the desired result. For example, if the script called 
for the selection of Candidate A, then the tester would keep touching the 
screen until Candidate A was selected. A review team monitored the 
testing to ensure that (1) the proper candidate for the Florida-13 
congressional race was ultimately selected and (2) the review screen 
showed this candidate selection when it was first presented. 

As with the ballot test, we used the Unity election reporting manager to 
read the compact flash cards and processed the audit data or each ballot 
test machine. We generated the ballot image log and examined the 
individual test ballots in the ballot image log. We looked for the unique 
identifier that was used for each test ballot and then confirmed that the 
ballot image reflected the correct selection for the Florida- 13 race as 
called for by the test ballot. After the testing had been completed, the 
expected results shown in the test ballot scripts were compared to the 
actual results contained in the ballot image log and the results tape using 
the same process discussed in the ballot testing methodology. 


Results of Calibration 
Testing 


The 39 ballots used in this test confirmed that the system correctly 
recorded the displayed vote for the Florida-13 contest. We also noted that 
the miscalibration clearly made the machines harder to use and during an 
actual election these machines would have probably been either 
recalibrated or removed from service once the voter brought the problem 
to the precinct’s attention, according to a Sarasota County official who 
observed the tests. 28 


28 Our review of the election day records identified t wo reported cases on election day 
where the miscalibration of the iVotronic DRE led to its closure and discontinued use for 
the rest of the day. 
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Figure 11 shows an example of effects of our misealibration efforts on the 
screen that is used to confirm the calibration results. Specifically, the 
stylus points to where the tester is touching the screen while the “X” on 
the screen shows where the machine indicated the stylus was touching the 
screen. 29 In a properly calibrated machine, the stylus and the “X” are 
basically at. the same point 


Figure 1 1 : Example of the Effects of a Miscaltferated Machine on the Calibration 
Screen 



Source: GAO. 


Figure 12 shows an example of where the tester is touching the screen to 
make a selection and how this “touch” is translated into a selection. As can 
be seen, the finger making the selection is touching a position that in a 
properly calibrated machine would not result in the selection shown. 
However, the machine clearly shows the candidate selected and our tests 
confirmed that for the 39 ballots tested, the candidate actually shown by 
the system as selected (in this example, the shaded line) was the candidate 
shown on the review screen, as well as the candidate that received the 
vote when the ballot was cast. 


29 While votes are normally cast using fingers on the touch screen, a stylus is normally used 
during the calibration process. 
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Source: GAO. 


Conclusions Our tests showed that (1) the firmware installed in a statistically selected 

sample of machines used by Sarasota County during the 2006 general 
election matched the firmware certified by the Florida Division of 
Elections, and we confirmed that when the manufacturer rebuilt the 
iVotronic 8.0. 1.2 firmware from the escrowed source code, the resulting 
firmware matched the certified version of firmware held in escro w, (2) the 
machines properly displayed, recorded, and counted the selections for all 
test ballots cast during the ballot testing involving the 1 12 common ways a 
voter may interact with the system to cast a ballot for the Florida- 13 race, 
and (3) the machines accurately recorded the test ballots displayed on 
deliberately miscaiibrated machines. The results of these tests did not 
identify any problems that would indicate that the iVotronic DREs were 
responsible for the undervote in the Florida-13 race in the 2006 general 
election. 

As we noted when we proposed these tests, even after completing these 
tests, we do not have absolute assurance that the iVotronic DREs did not 
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play any role in the large undervote. Absolute assurance is impossible to 
achieve because we are unable to recreate the conditions of the election in 
which the undervote occurred. Although the test results cannot be used to 
provide absolute assurance, we believe that these test results, combined 
with the other reviews that have been conducted by Florida, GAO, and 
others, have significantly reduced the possibility that the iVotronic DREs 
were the cause of the undervote. At this point, we believe that adequate 
testing has been performed on the voting machine software to reach this 
conclusion and do not recommend further testing in this area. Given the 
complex interaction of people, processes, and technology that must work 
effectively together to achieve a successful election, we acknowledge the 
possibility that the large undervote in Florida’s 13th Congressional District 
race could have been caused by factors such as voters who intentionally 
undervoted, or voters who did not properly cast their ballots on the 
iVotronic DRE, potentially because of issues relating t o interaction 
between voters and the ballot. 


Comments 


We provided draft copies of this statement to the Secretary of State of 
Florida and ES&S for their review and comment. We briefed the Sarasota 
County Supervisor of Elections on the contents of this statement and 
asked for their comments. The Florida Department of State provided 
technical comments, which we incorporated. ES&S and the Sarasota 
County Supervisor of Elections provided no comments. 


Mr. Chairman, this completes my prepared statement. I would be happy to 
respond to any questions you or other Members of the Task Force may 
have at this time. 
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Appendix I: Methodology for Selecting 
IVotronic DREs for GAO Testing 


Each of the three tests — firmware verification, ballot, and calibration — 
was conducted on a sample of the 1,499 iVotronic DREs that recorded 
votes during the 2006 general election in Sarasota County, Florida. We 
selected 1 15 iVotronic DREs for the firmware test, 10 for the ballot test, 
and 2 for the calibration test. Appendix II contains the serial numbers of 
the iVotronic DREs that were tested. 


Firmware Test Sample We selected a stratified random probability sample of iVotronic DREs 

from the population of 1,499. The sample was designed to allow us to 
generalize the results of the firmware sample to the population of 
iVotronic DREs used in this election. We stratified the population into two 
strata based on whether the machines had been sequestered since the 2006 
general election. There were a total of 818 machines that were sequestered 
and 681 machines that had been used in subsequent elections. The 
population and sample are described in table 3. 

We calculated the sample size in each stratum using the hypergeometric 
distribution to account for the relatively small populations in each 
stratum. We determined each sample size to be the minimum number of 
machines necessary to yield an upper bound of 7.5 percent, at the 99 
percent confidence level, if we observed zero failures in the firmware test. 
Assuming that we found no machines using an uncertified firmware 
version, these sample sizes allowed us to conclude with 99 percent 
confidence that no more than 7.5 percent of the machines in each stratum 
were using uncertified firmware. Further, this sample allowed us to 
conclude that no more than 4 percent of the 1,499 iVotronic DREs were 
using uncertified firmware, at the 99 percent confidence level. 


Table 3: Description of the Stratified Population and Sample Sizes for the Firmware 
Test 

Stratum 

Population size 

Sample size 

Sequestered machines 

818 

58 

Non-sequestered machines 

681 

57 

Total 

1,499 

115 


Source: GAO based on analysis ol Sarasota County voting data. 


An additional five sequestered machines and five non-sequestered 
machines were selected as back-up machines should there be problems in 
locating the selected machines or some other problem that prevented 
testing them. 
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Appendix I: Methodology for Selecting 
IVotronic DREs for GAO Testing 


Ballot Test Sample We randomly selected a total of 10 machines from the population of 1,384 

machines that were not selected in the firmware test sample. This sample 
size is not sufficient to allow us to make direct generalizations to the 
population. However, if we are reasonably confident that the same 
software is used in all 1,499 machines, then we are more confident that the 
results of the other tests on a small number of machines can be used to 
obtain increased assurance that the iVotronic DREs did not cause the 
undervote. We randomly selected one machine from each of the nine 
ballot styles used during the general election and one machine from the 
machines used for early voting. 1 In case of problems in operating or 
locating the machines, we also selected randomly selected two additional 
machines for each ballot style and for early voting. 


Calibration Test Sample The two iVotronic DREs selected for calibration testing were selected 
from those tested in the ballot test. Because the machines used for the 
ballot tests included an ADA machine and “standard” machines, we 
selected one of each for calibration testing. Although we did not test the 
ADA capabilities of the ADA machine (e.g., the audio ballots), we found 
that the on-screen appearance of selections on the ADA machine differed 
slightly from that on non-ADA machines. For example, the standard non- 
ADA machine displayed a blue bar across the screen and an X in the box 
next to the candidate’s name when a selection was made, while an ADA 
machine only showed an X in the box next to the candidate’s name. 


1 We also excluded those election day machines from one precinct that, support ed two 
different, ballot styles. 
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Appendix II: List of Machines Tested by GAO 


Table 4 table lists the iVotronic DREs that were tested by GAO. For each 
machine, the table shows whether the machine was sequestered and what 
type of testing was conducted on the machine. 


Table 4: List of iVotronic DREs Tested by GAO 


Serial number 

Machine 

sequestered 

Type of 

testing conducted 

V0105178 

No 

Firmware testing 

V01 05203 

No 

Firmware testing 

VO 105222 

Yes 

Firmware testing 

VO 105255 

No 

Firmware testing 

V01 05305 

No 

Firmware testing 

V0105351 

No 

Firmware testing 

V0 105379 

Yes 

Firmware testing 

V01 05390 

Yes 

Firmware testing 

V01 05396 

No 

Firmware testing 

V01 05422 

Yes 

Firmware testing 

V0 105481 

No 

Firmware testing 

V01 05499 

No 

Firmware testing 

V0 105500 

Yes 

Firmware testing 

V0 105524 

No 

Firmware testing 

V01 05526 

Yes 

Firmware testing 

V0 105563 

No 

Firmware testing 

V01 05573 

No 

Firmware testing 

V01 05607 

No 

Firmware testing 

V0105613 

Yes 

Firmware testing 

V01 05623 

Yes 

Firmware testing 

V0 105651 

No 

Firmware testing 

V01 05656 

No 

Firmware testing 

V0 105661 

Yes 

Firmware testing 

V01 05664 

Yes 

Firmware testing 

V0 105743 

No 

Firmware testing 

V01 05848 

No 

Firmware testing 

V0 105873 

Yes 

Firmware testing 

V01 05874 

No 

Firmware testing 

V0 105894 

Yes 

Firmware testing 

V01 05903 

Yes 

Firmware testing 

V01 05906 

Yes 

Firmware testing 
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Appendix II: List of Machines Tested by GAO 


Serial number 

Machine 

sequestered 

Type of 

testing conducted 


V01 05923 

Yes 

Firmware testing 


V01 05964 

Yes 

Firmware testing 


VO 105971 

Yes 

Firmware testing 


V01 05992 

Yes 

Firmware testing 


V01 06001 

Yes 

Firmware testing 


V0106016 

No 

Firmware testing 


V01 06024 

Yes 

Firmware testing 


V01 06025 

Yes 

Firmware testing 


V01 06034 

No 

Firmware testing 


V01 06064 

No 

Firmware testing 


V01 06068 

No 

Firmware testing 


VO 106069 

Yes 

Firmware testing 


V01 06084 

No 

Firmware testing 


V01 06087 

Yes 

Firmware testing 


V0106126 

No 

Firmware testing 


V0106156 

No 

Firmware testing 


V0106191 

Yes 

Firmware testing 


V01 06203 

Yes 

Firmware testing 


V01 06254 

Yes 

Firmware testing 


VO 106264 

Yes 

Firmware testing 


VO 106265 

No 

Firmware testing 


V01 06274 

No 

Firmware testing 


V01 06282 

No 

Firmware testing 


VO 106343 

No 

Firmware testing 


VO 106368 

No 

Firmware testing 


VO 106377 

No 

Firmware testing 


VO 106396 

Yes 

Firmware testing 


V01 06445 

No 

Firmware testing 


V0106461 

No 

Firmware testing 


V01 06475 

Yes 

Firmware testing 


V01 06478 

Yes 

Firmware testing 


V01 06486 

No 

Firmware testing 


V01 06507 

No 

Firmware testing 


VO 106522 

Yes 

Firmware testing 


VO 106525 

Yes 

Firmware testing 


V01 06531 

No 

Firmware testing 
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Appendix II: List of Machines Tested by GAO 


Serial number 

Machine 

sequestered 

Type of 

testing conducted 

V01 06552 

No 

Firmware testing 

V01 06585 

No 

Firmware testing 

V0106586 

NO 

Firmware testing 

V01 06588 

No 

Firmware testing 

V01 06602 

No 

Firmware testing 

V0106615 

Yes 

Firmware testing 

V01 06656 

Yes 

Firmware testing 

V01 06658 

Yes 

Firmware testing 

V01 06661 

No 

Firmware testing 

V01 06667 

Yes 

Firmware testing 

V01 06681 

No 

Firmware testing 

V0106711 

Yes 

Firmware testing 

V0106718 

Yes 

Firmware testing 

V01 06740 

No 

Firmware testing 

V0106744 

No 

Firmware testing 

VO 106833 

Yes 

Firmware testing 

V01 06840 


Firmware testing 

V01 06864 


Firmware testing 

V01 06865 


Firmware testing 

V01 06878 

Yes 

Firmware testing 

VO 106881 

Yes 

Firmware testing 

V01 06883 


Firmware testing 

VO 106907 

No 

Firmware testing 

V01 06933 


Firmware testing 

V01 06936 


Firmware testing 

VO 106949 


Firmware testing 

V01 06965 

Yes 

Firmware testing 

VO 107000 

No 

Firmware testing 

V010701 1 

No 

Firmware testing 

VO 107020 

No 

Firmware testing 

V01 07042 

Yes 

Firmware testing 

VO 107045 

No 

Firmware testing 

VO 107053 

Yes 

Firmware testing 

VO 107077 

Yes 

Firmware testing 

VO 107082 

No 

Firmware testing 

V01 07094 

Yes 

Firmware testing 
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Appendix II: List of Machines Tested by GAO 


Serial number 

Machine 

sequestered 

Type of 

testing conducted 

V01071G8 

Yes 

Firmware testing 

V0107138 

Yes 

Firmware testing 

V0107143 

No 

Firmware testing 

V0107147 

Yes 

Firmware testing 

V01 10355 

Yes 

Firmware testing 

V0111064 

No 

Firmware testing 

V0113816 

No 

Firmware testing 

V01 14087 

Yes 

Firmware testing 

V0114415 

Yes 

Firmware testing 

V01 17658 

No 

Firmware testing 

V01 18183 

No 

Firmware testing 

V01 18293 

Yes 

Firmware testing 

V01 05386 

Yes 

Early voting ballot testing 

V01 05266 

Yes 

Election day ballot testing 

VO 105694 

No 

Election day ballot testing 

V01 06082 

Yes 

Election day ballot testir 

V0106145 

Yes 

Election day ballot testing 

V01 06247 

Yes 

Election day ballot testing 

V01 06509 

No 

Election day ballot testing 
and calibration testing 

V01 06671 

Yes 

Election day ballot testing 

V01 17861 

NO 

Election day ballot testing 
and calibration testing 

V01 17951 

No 

Election day ballot testing 


Source: GAO. 
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The Chairman. The Chair is going to recognize himself for some 
preliminary questions and then, of course, turn it over to my col- 
leagues. 

But, Dr. Barkakati, I have a couple of questions. First of all, that 
you were specifically charged with simply looking as to whether the 
electronic machines malfunctioned and may have been the culprit 
in the undervote; is that correct? 

Mr. Barkakati. Yes, sir. 

The Chairman. We did not ask you to look into ballot design? 

Mr. Barkakati. No, we did not look at the ballot design. 

The Chairman. And we did not ask you to go and look into either 
voter apathy or just whether they were totally turned off by the 
tenor of the campaign and decided to intentionally or deliberately 
not vote in Florida- 13; is that correct? 

Mr. Barkakati. Yes, that is correct. 

The Chairman. In conducting your testing and the protocols — be- 
cause, of course — again, for just general information, you came be- 
fore the task force with your intended protocols as to how you were 
going to conduct your testing. And in arriving at that protocol, did 
you entertain suggestions, recommendations, basic input from all of 
the parties, including the contestant, the contestee, Florida election 
officials, Florida State University personnel that conducted some 
previous testing and their experts, as well as suggestions and rec- 
ommendations or observations from the manufacturer of the voting 
machine? 

Mr. Barkakati. Yes, sir, we did entertain all of those inputs in 
deciding the test protocols and procedures. 

The Chairman. Okay. Now, there are many individuals that are 
interested in this for a lot of reasons. Because it goes beyond this 
particular Florida- 13 contested election. And as much as I’d like to 
just contain everything to the question at hand and which will ba- 
sically form the basis for everything that we do — we are not going 
to expand it. However, we do have individuals out there that have 
expressed great interest; and I have one individual who e-mailed 
some information to my office this morning. And so I do want to 
go ahead and ask some of the questions that they — that he has ac- 
tually posed. 

And one would be, the personalized electronic ballots, or PEB 
cards, used to activate each voter’s ballot for the DRE, the elec- 
tronic voting machine, contains firmware separate from the 
firmware for the DRE itself. And this individual believes that that 
could have been a culprit, that could have been — given you some 
insight as to whether the machines malfunctioned. 

Outside of that which you may have tested specifically, how 
would you address this individual’s concern regarding the PEB 
card which you cover extensively in your final report? 

Mr. Barkakati. Sir, the PEB, personalized electronic ballot, I 
mean, does have a little bit of capability to transmit back and forth 
information with the iVotronic machine using infrared communica- 
tion mechanism. However, besides that, it is primarily a memory 
device to hold data; and it holds initially the ballot definitions that 
are going to be used to display the ballots on the screen. However, 
when you open the machine, you copy all of those into the iVotronic 
and then from that point on the PEB is only an activator device 
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and it can activate the iVotronic to get started. So there were some 
allegations that maybe the PEB was displaying different ballots to 
different people. 

However, it was copied once, and then the same ballot displays 
over and over again. And we have seen that every machine that 
was used in all the 157 precincts had votes cast on the Florida-13 
race. So we could not have had a situation where it was selectively 
not displaying ballots. 

I mean, we have checked into that part, that the PEB could not 
have been the cause of the problem, including the fact that the 
PEB is not the one that does the primary task of the calculations 
or the displaying of the ballot. It is all done in the firmware that 
is inside the iVotronic machine itself. 

As I say, the PEB does have a small capability to send things 
back and forth and that is all, to the extent it does some work. So 
what I am saying is that we looked at the whole situation that the 
PEB could be a cause — both us as well as the Florida State Univer- 
sity review team that looked at, you know, the same scenarios — 
and we concluded that it could not be the reason for the undervote. 

The Chairman. All right. 

And, my second question, again taken from this particular indi- 
vidual — and I am going to read specifically from the e-mail: “It is 
conceivable that — and even likely that were there bugs in the user 
interface in the Sarasota machines such as intermittent smoothing 
filter problems or other anomalies. These: (A) Could have occurred 
regardless of whether the software, firmware were identical; and, 
(B) would not have been uncovered without greater volume testing 
with actual use of the interface, especially in conditions resembling 
election day use.” 

Can you respond? 

Mr. Barkakati. Yes, sir. 

We actually — as you might recall, there was some mention that 
the smoothing filter — in one of those letters that came from, I 
think, ES&S to the county saying that we have a smoothing filter 
that we want to change and, you know, it will help with the per- 
formance of the machine in terms of responsiveness. 

Now, the source code review team in Florida, they — they said 
they did not find the smoothing filter. However, when we were 
looking at it — because we already knew that, you know, was an 
issue coming up, we could find certain elements that I was able to 
confirm with the manufacturer that that constituted what they 
were calling a smoothing filter. 

It essentially is very simple to explain. Specifically, the machine 
does not give you — as soon as you touch it, it does not assume that 
you have touched it yet. It records the point and then it waits a 
little bit, like 200 milliseconds or so, and then tries to get another 
data point to see that both are close enough. And if they are not 
close enough, then they assume that, one, maybe it is a spurious 
thing and then I would ignore it. And that is the logic built in. And 
that logic is what was called smoothing filter. 

Now, how close the two touches have to be is the threshold that 
they can play with to get it to be, like, more sensitive, less sensitive 
kind of thing. If it is like wide — as you can imagine, the threshold 
is wide — I mean, it doesn’t matter if another touch is slightly far- 
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ther off, it still assumes them to be the same. Whereas if you tight- 
en them up, it takes longer, because the touch screens do have 
some noisy, you know, data going back and forth. 

So, specifically, the bottom line was the smoothing filter was not 
more of a mystery after that. I can understand why it was there, 
and we figured it out, and the proposal they had made was to sim- 
ply to widen the threshold to make it more responsive. 

Regardless of all of that, we were looking at a source code that 
handles the display and the recording of the vote. That is not going 
to be affected at all by the filter — other than the machine’s re- 
sponse might be slow. In other words, it might take a little time 
to get the mark to show up. 

But the machine isn’t going to suddenly misbehave because it is 
slightly slower in response, and that actually almost explains the 
other element to which you might have read, like nondeterministic 
behavior. 

Well, because the touch screen — I mean, depending on the time, 
some of the conditions — it may, you know, have other spurious 
things going on whereby that filter that I talked about, the close- 
ness of two touches, maybe it’s not being met. And if it is not being 
met, the condition is not met, then the system waits a little bit 
longer to get another touch, you know. So it looks like it — is it 
waiting for me to get — you know, accept my input. That is what the 
user feels like. 

But the bottom line of all of that is, regardless of all of that, that 
it isn’t going to affect anywhere at all in terms of what selection 
is displayed to the user and what selection is recorded. From that 
we are concluding that, essentially, if the machine is displaying a 
selection, that will be the one displayed on the review screen and 
recorded when you press the vote button; and that is the bottom 
line that we are going after in the technological study. 

So I guess it is a very long way of explaining, but I think we did 
understand — we looked into all of that and we understood what it 
was, smoothing filter was, and what the company was intending to 
do when they sent the letter and what its effect might have been. 
And, regardless of all of that, we were able to confirm sort of like 
the same thing, that the source code review from — team from Flor- 
ida State University had said, it wasn’t going to affect the vote dis- 
play and recording of the machine. 

The Chairman. Thank you very much, Doctor. 

I will recognize Congressman McCarthy at this time for his ques- 
tions. 

Mr. McCarthy. Thank you, Mr. Chairman. 

In reading your report and listening to you when we first started 
out and talking to you about whether you could do this work and 
setting out some parameters to look at it, you told us that the clos- 
est statistically that anyone can get is 99 percent. Now, in hearing 
your report, you say you are 99 percent sure every single vote that 
was voted has been counted and counted correctly. Is that true? 

Mr. Barkakati. I guess it is a bit nuanced in the sense that the 
statistical portion of it applies to the — confirming that every ma- 
chine was running the same software. So that was — that does have 
a 99 percent confidence level, that most machines are running the 
same software. 
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The remaining part is based on a logic, essentially, kind of like 
an assembly line or something. When you have a lot of the widgets 
being built, you can take one and sample and it is okay because 
everybody is the same. Extending that logic, we are able to say 
that, while we test a small number of machines, they record votes 
properly and display things properly and therefore that applies to 
the whole population. And then taking the abundance of all the 
testing that has been done in the past by the State of Florida and 
others, we are basically forming the judgment which is what we 
say, that we collectively think everything has been done to elimi- 
nate the machine as the cause of the undervote. 

So, in a way, you are getting the answer, but the 99 percent 
number in a nuanced way applies only to the condition that the 
firmware is the same. But the rest should be — since that is our — 
basically and logically, as I explained, was the reason behind com- 
ing up with a conclusion that adequate testing has been done to 
eliminate the machine. So, essentially, you are getting the conclu- 
sion without the 99 percent number associated with it because stat- 
isticians are not going to let me say it that way. 

Mr. McCarthy. So in my world we say 100 percent, but I under- 
stand. 

And I will tell you — and having been an individual that worked 
on staff on some other contested races, this one has been more 
thoroughly investigated than any one I have seen. You had an 
independent Florida State review prior to it ever coming to Con- 
gress’ review, and they were — we even had some of the individuals 
that did the research on that before us testify at other times. And 
I found that to be very thorough and very correct. 

In reading your report, you even said you miscalibrated the ma- 
chines, and they still worked properly. 

Mr. Barkakati. Yes. The miscalibration was really meant to be 
a small amount and a larger amount of miscalibration to see the 
effect of it, essentially. And what we observed is that — and wanted 
to confirm that, even if miscalibrated, if it is showing you some- 
thing as the selection you are making, then that would be the se- 
lection that will eventually get recorded in the memory inside the 
machine. That is what our, you know, goal was. 

And we did find that all of the machines, even with a smaller 
and larger amount of miscalibration — of course, it gets hard to use 
and they are so obvious in terms of bigger miscalibrations, become 
so obvious that — you probably will — officials have told us that they 
would probably have found that right away and put it out of serv- 
ice in any case. 

But, regardless — and coming back to your question — we did do 
that, and we found that in each case they definitely were dis- 
playing — if you can make any selection on the screen, that is the 
selection that is going to be essentially your choice that will be re- 
corded. 

So we confirmed that, yes. 

Mr. McCarthy. That is very interesting, really, to the American 
public, too. Because Mr. Ehlers always points out to us, even when 
you went back and you just did paper and you would recount, you 
would always find one or two differences because you had human 
error touching it a number of times. And here we had individuals 
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researching numerous different ways in the computer, and it came 
out where no vote was different, and it all worked properly. So that 
is nice to know. 

Because I think our goal here was set out first when the chal- 
lenger came in and wanted the election overturned and we went 
through this research here. We have found time and time again 
that every vote in the 13th district of Florida was counted properly 
and was put forward. So that is really a testament to the American 
public and to this district that they know their election was honest, 
true and correct. And I applaud the work you have done and the 
work that the others have done prior because we have never re- 
searched it this thoroughly, and I thank you. 

The Chairman. Thank you very much, Mr. McCarthy. 

The Chair will recognize Ms. Lofgren. 

Ms. Lofgren. Mr. Chairman, you asked the questions that I was 
going to ask, so I am happy for that. 

I would just note that on page 12 of the report the GAO does 
make a suggestion that a voter-verified paper trail could provide an 
independent confirmation. And, obviously, we can’t create that 
retroactively. But, looking forward, I think that is something that 
we hope that the jurisdictions will look to do; and we will be hope- 
fully in the near future pursuing that kind of endeavor at the Fed- 
eral level. 

So, as with the other members of the task force, I — there was a 
big undervote. We don’t know why. I will say that when I looked 
at the replica of the ballot on page 10 I didn’t notice, because of 
the way it is set out, that there was actually an election above the 
State. So I — we will never know. But I think that that had an im- 
pact here. 

I think that it is important that we have discharged our duty 
here with the help of GAO in a way that has been nonpartisan 
really. Because our obligation is not as Democrats or Republican. 
Our obligation is to find out as best we can what happened and to 
make a decision based on that alone. And I think that is what we 
have done, and that is what we should have done. And I feel very 
satisfied with the process, and I am eager to put this behind us 
and get on to other business. 

So I don’t know if there are other questions. At the appropriate 
time, I have a motion to make. 

The Chairman. Thank you very much, Ms. Lofgren. 

And the Chair will recognize Mr. Ehlers, the ranking member of 
the full committee. 

Mr. Ehlers. Thank you, Mr. Chairman; and I will be very brief. 

First of all, I just want to thank the GAO that did a very thor- 
ough job under very difficult circumstances. I commend you for 
your work. 

The other comment I would make is to agree with Ms. Lofgren 
about the ballot design. I had exactly the same experience. When 
I looked at the ballot, I actually missed the congressional race. I 
think this is a valuable experience, too, in terms that has been 
transmitted nationwide. I think every county clerk, city clerk, 
township clerk, and Secretary of State will be looking at ballot de- 
sign more carefully in the future. 

With that, I yield back. 
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The Chairman. Thank you very much, Mr. Ehlers; and the Chair 
recognizes Mr. Lungren. 

Mr. Lungren. Thank you, Mr. Chairman. 

Dr. Barkakati, I know statisticians don’t like to say certain 
things and so forth, but is it fair to say — to summarize in part 
what you said, that based on the work that your organization has 
done and based on your review of those other organizations that 
did testing, you do not see a need for any additional testing or are 
not suggesting to this committee that we request any additional 
testing to answer the questions you were requested to answer? 

Mr. Barkakati. Yes, I agree with your statement. We are not 
suggesting any further testing based on all the things we have seen 
so far and done so far. 

Mr. Lungren. I thank you. And I join my colleagues in sug- 
gesting that the ballot design is something that needs to be looked 
at. 

But I also may be the only one willing to confess that I have gone 
to the ballot at times and intentionally not voted in certain elec- 
tions to show my protest over the way the campaign was con- 
ducted. And I just note that this was a hard-fought campaign not 
only in the general, but in the primary, and voters sometimes re- 
spond in that way. And I think we ought to recognize — or at least 
I will admit I have been one that has done that. And I have been 
in similar situations. Maybe you would be investigating as to why 
I didn’t vote. 

But I thank the chairman once again for the professional way in 
which he has handled this and all the members of the task force. 

The Chairman. Thank you very much, Mr. Lungren. 

Now I am going to finish up with a couple of questions, one that 
was actually posed by a representative for the contestant. And even 
though this was not your charge, and we understand that. And I 
don’t want to get too far afield. But there have been some — obvi- 
ously, we all have our take on what may have happened, ballot de- 
sign or simply [a] voter that was simply turned off. Probably more 
ballot design, if we look at it somewhat objectively. 

And the reason for that is this only — the undervote was experi- 
enced in this degree — or extent in Sarasota. So it couldn’t be just 
the apathy or the intolerance or the disgust stopped at the county 
line and — nor was it reflected, I believe, in the absentee voting or 
those votes that were cast by methods other than the ES&S ma- 
chine. 

So the question is as follows: Has the GAO reached any conclu- 
sions about whether the unusually elevated undervote rate was due 
to intentional undervoting or to unintentional human factors such 
as voter confusion caused by poor ballot design? 

Mr. Barkakati. Unfortunately, we didn’t do anything to deter- 
mine between those two what might be the reason, except for 
knowing the other explanations. People have stated that ballot de- 
sign might be an issue. We are aware of those kind of things. But 
we did not really evaluate to kind of try to eliminate one versus 
the others. So, unfortunately, I don’t think we have any conclusive 
statement about whether it was the ballot design that might have 
caused it or the intentional undervoting that caused it. That is 
probably all 
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I mean, I am aware of other — we have mentioned in a past state- 
ment that there were other, I think, experiments being done on hu- 
mans — using humans to see — or human subjects to see whether 
they can miss it or not by, for instance, a professor at MIT that 
we know of. Ted Selker is the name. And then, for that one, we 
know that the work is proceeding on that area to find out whether 
ballot design might have been the reason, but he hasn’t finished 
the whole results, I guess, yet. And he was working with ES&S 
machines actually, using the same ballot layout and all. So there 
may be some report results coming out from that in the future 
which might provide, you know, more light into this area. 

The Chairman. Because I think people still want some answers. 
And, of course, again, that wasn’t the central allegation that we 
had to resolve here before the committee. 

One last observation, of course, is, this really does point out — it 
goes back to ballot design. It doesn’t matter how sophisticated and 
reliable the voting machine may be. It could even have a paper 
trail. But at the end of this whole process is that ballot design 
many times can be confusing. And we know that Florida has more 
or less been the poster child, but they have moved forward. But 
still, ballot design could still present a real, real problem as to 
whether it was the butterfly ballot in 2000 or whether it was the 
2006 Florida-13 election. 

So we understand the tremendous challenge that local election 
officials have; and we would just caution them to take, again, every 
precaution out there that they possibly could regarding how they 
designed that ballot for presentation. 

The last thing I want to say is that every candidate for office has 
a right, when they are running for the United States House of Rep- 
resentatives, to challenge the validity of the election if, obviously, 
they were not the victor. And that is their right. And then it is the 
constitutional duty of the United States Congress, then, to pass on 
whether someone is going to be seated and sworn in as a Member 
of the House. And I think that was our charge, that was our duty. 
We want to thank everyone that assisted us in performing that 
duty. 

At this time, I am going to recognize Ms. Lofgren for the purpose 
of making a motion. 

Ms. Lofgren. Mr. Chairman, I move that the chairman be au- 
thorized and directed to report to the committee that the task force 
has completed its investigation related to the election of a rep- 
resentative from the 13th Congressional District of Florida to the 
House of Representatives, and I move further that the chairman 
report to the committee the task force’s recommendation that the 
election contest in the 13th District of Florida be dismissed. 

[The information follows:] 
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February 8, 2008 

FL-13 Task Force Motion #7 

(Offered by Zoe Lofgren) 

DISMISSING THE ELECTION CONTEST IN THE 
THIRTEENTH CONGRESSIONAL DISTRICT OF FLORIDA 


I move that the Chairman be authorized and directed to report to the Committee that the 
Task Force has completed its investigation related to the election of a Representative 
from the 13 lh Congressional District of Florida to the House of Representatives, and I 
move further that the Chairman report to the Committee the Task Force's 
recommendation that the election contest in the 13 th District of Florida be dismissed. 
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The Chairman. And at this time, task force members, if they are 
in favor, will register by signaling aye. Opposed, nay. 

It’s unanimous, the ayes; and we will proceed with this formal 
adoption of the motion and report it to the full committee for its 
consideration at a later date. 

And, with that, we stand adjourned. 

[Whereupon, at 10:49 a.m., the briefing was adjourned.] 



